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Abstract. Let A 2 be a free associative or polynomial algebra of rank 
two over a field K of characteristic zero. Based on the degree estimate 
of Makar-Limanov and J.-T.Yu, we prove: 1) An element p G A 2 is a 

C"""- test element if p does not belong to any proper retract of A 2 ; 2) Ev- 

ery endomorphism preserving the automorphic orbit of a nonconstant 

<£ element of A 2 is an automorphism. 

43 ■ l. Introduction and main results 

a 

In the sequel, K always denotes a field of of characteristic zero. 
Automorphisms (endomorphisms) always mean i^-automorphisms (K- 
endomorphisms) . 

(N 

Let A n be a free associative or polynomial algebra of rank n over 
K. An element p G A n is called a test element if every endomorphism 
of A n fixing p is an automorphism. A subalgebra R of A n is called a 
retract if there is an idempotent endomorphism 7r(7r 2 = 7r) of A n (called 
a retraction or a projection) such that 7r(A n ) = i?. Test elements and 
retracts of groups and other algebras are defined in a similar way. Test 
elements and retracts of algebras and groups have recently been studied 

W 

A test element does not belong to any proper retract for any algebra 

or group as the corresponding non-injective idempotent endomorphism 

is not an automorphism. The converse is proved by Turner [51] for free 

groups, by Mikhalev and Zolotykh [24] and by Mikhalev and J. -T. Yu 

[2TI [22] for free Lie algebras and free Lie superalgebras respectively, 



o 
oo 
o 
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and by Mikhalev, Umirbaev and J. -T. Yu [19] for free nonassociative 
algebras. See also Mikhalev, Shpilrain and J. -T. Yu [18]. 

In view of the above, we may raise the following 

Conjecture 1. If an element p G A n does not belong to any proper 
retract of A n , then p is a test element. 

Recently, V. Shpilrain and J. -T. Yu [33] proved Conjecture 1 for 
C[x, y\. A key lemma in their proof is the degree estimate of Shestakov 
and Umirbaev [26], which plays a crucial role in the recent celebrated 
solution of the Nagata conjecture [27J [28] and the Strong Nagata con- 
jecture [35] . 

More recently, Makar-Limanov and J. -T. Yu [16] developed a new 
combinatorial method based on the Lemma on radicals and obtained a 
sharp degree estimate for the 'free' case, namely, for a free associative 
algebra or a polynomial algebra over a field of characteristic zero. It has 
found applications for automorphisms and coordinates of polynomial 
and free associative algebras. See S.-J. Gong and J.-T. Yu [9]. 

Now we consider another related problem. In an algebra or a group, 
certainly an automorphism preserves the automorphic orbit of an ele- 
ment p. The converse is proved by Shpilrain [31] and Ivanov [TU] for 
free groups of rank two, by D. Lee [2] for free groups of any rank, 
by Mikhalev and J. -T. Yu [22J for free Lie algebras and by Mikhalev, 
Umirbaev and J.-T. Yu [19] for free non-associative algebras, by van 
den Essen and Shpilrain [7] for A 2 when p is a coordinate, by Jelonek 
[TT] for polynomial algebras over C when p is a coordinate. For the 
related linear coordinate preserving problem, see, for instance, S.-J. 
Gong and J.-T. Yu [8]. See also the book 



In view of the above, we may raise the following 

Conjecture 2. Let p e A n — K. Then any endomorphism of A n 
preserving the automorphic orbit of p must be an automorphism. 

Conjecture 2 has recently been settled affirmatively by J.-T. Yu [36] 
for A2 = <C[x,y] based on Shpilrain and J.-T.Yu's characterization of 
test elements of C[x,y] in [33] and the main result in Drensky and 
J.-T.Yu 0. 
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In this paper, based on the recent degree estimate of Makar-Limanov 
and J.-T.Yu [IS], the main ideals and techniques in Drensky and J.- 
T.Yu [6], Shpilrain and J.-T. Yu [32], [33], and J.-T.Yu [36], we prove 
both Conjecture 1 and Conjecture 2 for n — 2. Our main results are 

Theorem 1.1. If an element p G A 2 does not belong to any proper 
retract of A 2 , then p is a test element of A 2 . 

Theorem 11.11 was proved by Shpilrain and J.-T.Yu [33] for A 2 = 
C[x,y]. 

Theorem 1.2. If an endomorphism <fi of A 2 preserves the automorphic 
orbit of a nonconstant element p G A 2 , then <fi is an automorphism of 
A 2 . 

Theorem 11.21 was proved by J.-T.Yu [36J for A 2 = C[x,y}. 

Crucial to the proofs of the above two theorems are the following 
two results, which have their own interests. 

Theorem 1.3. Let p G A 2 have outer rank two. Then any injective 
endomorphism <fi of A 2 is an automorphism if <f)(p) = p. 

Theorem 1 1 . 3 1 may be viewed as an analogue of a result in Turner [34J 
for free groups. It was proved for A 2 = C[x, y] in J.-T.Yu [36] based on 
a result in Shpilrain and J.-T.Yu [33] . 

Theorem 1.4. An element p(x , y) G A 2 belongs to a proper retract of 
A 2 if p(x,y) is fixed by a noninjective endomorphism (p of A 2 . More- 
over, in this case there exists a positive integer m such that m is a 
retraction of A 2 . 

Theorem 11.41 was proved for A 2 = C[x,y] in Drensky and J.-T.Yu 
®- 

2. Proofs 

The following two lemmas are Theorem 1.1 and Proposition 1.2 in 
Makar-Limanov and J.-T.Yu [TBI. 



Lemma 2.1. Let A n = K(xi,--- ,x n ) be a free associative algebra 
over a field K of characteristic zero, f,gEA be algebraically indepen- 
dent, f + and g + are algebraically independent, or f + and g + are al- 
gebraically dependent and neither deg(/) | deg(g) nor deg(g) \ deg(f), 
p G K(x,y) . Then 

deg(p(f,g)) > d ^ '% deg(/),do g ( fl )(p)- 
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Here deg is the total degree, Wdeg(/),deg(g)(p) is the weighted degree of 
p when the weight of the first variable is deg(/) and the weight of the 
second variable is deg(g), / + and g + are the highest homogeneous com- 
ponents of / and g respectively, and [/, g] = fg — gf is the commutator 
of / and g. 

Lemma 2.2. Let A n = K[xi, ■ ■ ■ ,x n ] be a polynomial algebra over a 
field K of characteristic zero, f,g G A be algebraically independent, 
p G K[x, y\. Then 

deg(p(f,g)) > 

(deg(f),deg(g))(deg(fg) - deg(J(f,g)) - 2) 1 



^deg(/),deg( ff )(p)[l 



deg(/) deg(g) 



Here deg is the total degree, Wdeg(/),deg( 9 )(p) is the weighted degree 
of p when the weight of the first variable is deg(/) and the weight of 
the second variable is deg(g), (deg(/), deg(g)) is the greatest common 
divisor of deg(/) and deg(g), deg(J(f,g)) is the largest degree of non- 
zero Jacobian determinants of / and g with respect to two of Xi, ■ ■ ■ ,x 



n ■ 



The following characterization of a proper retract of A2 was obtained 
by Shpilrain and J.-T.Yu [32] based on a result of Costa [3]. 

Lemma 2.3. Let R be a proper retract of A 2 . Then R = K[r] for some 
r G Ai. Moreover, there exists an automorphism a of A2 such that 
a(r) = x + w(x,y), where w(x,y) belongs to the ideal of A 2 generated 

byy- 

Lemma 2.4. Let p G A 2 with outer rank 2 and f, g G A n . Then 
w deg(f),deg( g )(p) > deg(/) + deg(g) . If every monomial of p contains 
both x and y and deg(p) > 2, then u>deg(/),deg(g) (p) > deg(/) + deg(g). 

Proof. 1) If p contains a monomial containing both x and y, where 
J/Oj'^O, Wdeg(/),deg(fl)(p) > *(deg(/))+j(deg(^)) > deg(/) + deg(^). 
If every monomial of p contains both x and y and deg(p) > 2, then the 
second inequality becomes strict. 

2) Otherwise p must contain monomials x 1 and yi where i > 2, j > 2. 
Then Wde g (/),de g ( ff ) (p) > 2max{deg(/), deg(5f)} > deg(/) + deg(flr). D 

Lemma 2.5. Let A n = K(x±, ■ ■ ■ , x n ) be a free associative algebra over 
an arbitrary field K of zero characteristic, f,g G A 2 be algebraically 
independent, p G K(x, y) have outer rank two. Then 

deg(p(f,g)) > deg[f,g}. 
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If every monomial of p contains both x and y and deg(p) > 2, then 

deg(p(/,#)) >deg[f,g]. 

Proof. Let 1) If / + and g + are algebraically independent; or / + , g + 
are algebraically dependent, but deg(/) \ deg(g) and deg(g) \ deg(/). 
Then by Lemma 12.11 and Lemma |2.4[ deg(p(/, g)) > deg[/, g\. If, in 
addition, every monomial of p contains both x and y and deg(p) > 2, 
then by Lemma I2TT1 and Lemma EH deg(p(/, g)) > deg[/, g\. 

2) Otherwise there exists an automorphism a, which is the compo- 
sition of a sequence of elementary automorphisms, such that oe(f) = 
f, a(g) — g,p — o>~ l {p) satisfying the condition in 1). Then deg(p(/, g)) 
= deg(p(/, g)) > degj/, g] = deg[/, g\. D 

Lemma 2.6. Let A n = K[x\, • • • ,x n ] be a polynomial algebra over 
an arbitrary field K of zero characteristic, f,g G A n be algebraically 
independent, p e K[x, y] has outer rank two. Then 

deg(p{f,g))>deg(J{f,g))+2. 

Proof. We may assume deg(/) = m, deg(g) = n. As p has outer rank 
2, by Lemma 12.41 then p contains a monomial with both x and y, or 
contains monomials x % and y- 7 where % > 2, j > 2. 

1) Let / + and (? + be algebraically independent. 

a) If there exists a monomial in p containing both x and y, then 
deg(p(f,g)) > deg(f) + deg(g) > deg(J(f,g)) + 2; 

b) Otherwise p must have a monomial of x* where i > 2, and another 
monomial j/ J where j > 2, then deg(p(/, p)) > 2 max{m, n} > deg(/) + 
deg(^)>deg(J(/,^)) + 2; 

2) Let / + , (? + be algebraically dependent, and m\n and n\m. 

c) If Wdeg(/),deg( 9 )(p) < lcm(m,n), then in p(f,g), f + and g + cannot 
cancel out, hence similar to the case 1 a), deg(p(/, g)) > deg(/) + 
deg(^)>deg(J(/^)) + 2. 

d) Otherwise Wdeg(/),deg(g)(p) >lcm(m, ra) = mn/(m,n). We also 
have mn = (m, n)lcm(m, n) > (m, n)(m + n). Hence deg(p(f,g)) > 
deg(J(f,g)) + 2 by Lemma O 

3) Let / + , g + be algebraically dependent, but m \ n or n \ m. Then 
by same process in the Proof 2) of Lemma 2.4, we may reduce to the 
above case 1) or case 2). □ 

Lemma 2.7. Let = (f,g) be an injective endomorphism of K(x,y) 
but not an automorphism. Then deg([(p k (x) , (j) k (y)}) > k + 2 for k > 0. 



6 SHENG-JUN GONG AND JIE-TAI YU 

Proof. We use induction. deg[0°(x), </>°(y)] = deg[x,y] = + 2. Assum- 
ing deg^" 1 (a;), fe ~%)] > (fc-l)+2. Define p(x,?/) := [f(x,y),g(x,y)]. 
As = (/, g) is injective, every monomial of p(x, y) contains both x and 
y. Since = (f, g) is not an automorphism, by the well-known result of 
Dicks (see, Dicks [4], or Cohn [2]), deg(p(x, y)) > deg(x) + deg(y) = 2. 
Applying Lemma |2~5| deg(p(u,v)) > deg[u,v] for u = fc_1 (x), v = 
4> k ^ 1 (y), hence 

deg[0 fe (x),0 fc (y)] = deg{p{(p k - 1 {x),(f) k - 1 {y))) > deg^" 1 ^), fc " 1 (y)] > 
(fc-l)+2 = jfc+l. Therefore, deg[0 fc (x), <j) k {y)\ > (fc+l)+l = k+2. D 

Lemma 2.8. Let <fi = (f,g) be an injective endomorphism of K[x,y] 
but not an automorphism and there exists an element p e K[x,y] fixed 
by 4>. Then deg( J((p k (x), 4> k (y))) > k for k > 0. 



Proof. As fixes p, is not an automorphism, by a result of Kraft 
(see also Shpilrain and J.-T.Yu [32]), deg(J(0(x), 4>{y))) = deg(J(f,g)) 

> 1. By the chain rule for the Jacobian, 

deg(J((f) k (x),(j) k (y))) 

= deg(J(f\g)(<p k -\x)A k -\y))(J(<i ) k ~ l (x),<p k -\y)))) 

> deg(J(0 fc - 1 (x),0^ 1 (y))) + 1. 



The proof is concluded by induction. 



□ 



Lemma 2.9. Let <fi = (f,g) be an injective endomorphism of A 2 but 
not an automorphism. Then any element p G A 2 with outer rank 2 
cannot be fixed by 0. 

Proof. If p e A 2 with outer rank two fixed by 0, then deg(p(/, g)) = 
deg(p(<fi k (x),(f) k (y))) > k + 2 for all k > 0. by Lemma [2.51 and Lemma 
12.71 for noncommutative case; and by Lemma 12.61 and Lemma 12.81 for 
polynomial case. The contradiction completes the proof. □ 

Proof of Theorem 11.31 

By Lemma E3JD 
Proof of Theorem 11.41 

The proof presented here is similar to the proof of the main Theorem 
in Drensky and J.-T. Yu [6]. 

Let p G A 2 — {0} fixed by a noninjective endomorphism of A 2 . Then 
0(x) and 4>(y) are algebraically dependent over K. Let us denote the 
image of 4>{A 2 ) by S = K[(j>(x) , <f>(y)] (since 0(x) and (p(y) are al- 
gebraically dependent, <f>(x) and (fi(y) are in a polynomial algebra of 
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rank one over K as a consequence of a result of Bergman pQ for non- 
commutative case and as a consequence of a result of Shestakov and 
Umirbaev [26] for polynomial case) and by Q(S) the field of fractions 
of S. Therefore the transcendence degree of Q(S) over K is 1. Let 
^ q(x,y) G (Ker(0)) n S 1 . Since p(x,y) also belongs to S 1 , the poly- 
nomials p and g are algebraically dependent and 

h(p, q) = a (q)p n + ai(g)p n_1 + . . . + a n _i(g)p + a n (q) = 

for an irreducible polynomial h(u,v) G K[u,v] and Oi(t) G -^[i], i = 
0, 1, . . . , n. Hence 000, q)) = h(<j){p), 0(g)) = h(p, 0), 

a (0)p n + a^p 71 - 1 + ... + a n _i(0)p + o n (0) = 0. 

Therefore ao(0) = Oi(0) = . . . = a n (0) = 0. Now the polynomials a.i(t) 
have no constant terms and h(u, v) is divisible by v which contradicts 
to the irreducibility of h(u,v). Therefore (Ker(0)) fl S = and <fi acts 
injectively on its image S. Hence we may extend the action of <fi on 
Q(S) (because axjb\ = a 2 /b 2 in Q(S) is equivalent to a0 2 = ^2^1 
and hence 0(ai/&i) = cj)(ai) / (j)(bi) = (j)(a 2 ) / </>&) = 0(02/^2))- By 
Liiroth's theorem (See, for instance, Schinzel [23]), Q(S) = K{w) for 
some w G Q(S). The automorphism fixes p(x,y) and its extension 
on QO) fixes K(p). Since w is algebraic over K(p), Q(S) is a finite 
dimensional vector space over K(p) and is a K(p)-lme&r operator 
of Q(S) with trivial kernel. Hence is invertible on Q(S) and we 
may consider as an automorphism of the finite field extension Q(S) 
over K(p) which fixes K(p). By Galois theory (0 interchanges the 
roots of the minimal polynomial of w over K(p) and there are finite 
number of possibilities for 0(w)), has finite order. Let m = 1. 
Then m+1 (r) = m (0(r)) = m (0(r)) = 0(r) for every r £ A 2 and 
O m ) 2 = m + 1 m " 1 = 00™- 1 = m Therefore vr = m is a retraction 
(idempotent endomorphism) of A 2 with a nontrivial kernel and ir(p) = 
p. Hence p(x, y) is in the image of 11 which is a proper retract ir(A 2 ) of 
A 2 . □ 

Proof of Theorem II. II 

As p G A 2 does not belong to any proper retract of A 2 , by Theorem 
I1.4I any endomorphism of A 2 fixing p must be injective. By Lemma 
obviously p must have outer rank two, otherwise p would belong to 



a proper retract of A 2 . By Theorem II. 3 \ is an automorphism. Hence 
p is a test element of A 2 . □ 

Proof of Theorem II .21 
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The proof presented here is similar to the proof of the main result 
Theorem 1.4 in J.-T. Yu 



We may assume that 4>(p) = p. By the definition of the test element, 
we may assume p is not a test element. By Theorem ll.il we may assume 
p belongs to a proper retract K[r] of A 2 . By a result in J.-T.Yu [36], we 
may assume p has outer rank 2. By Theorem II. 3 \ we may assume is 
non-injective. Suppose that p = f(r), where / G K[t] — K, deg(/) = m. 
By Theorem 11.41 tt = <p m is a retraction of A 2 to K[r]. As preserves 
the automorphic orbit of p, so does tt = <j) m . Applying Lemma 12.31 
(suppose a(r) = x + w(x,y), where w(x,y) £ K[y] belongs to the ideal 
of A 2 generated by y, a is some automorphism of A 2 , replace r by a(r), 
and tt by ana" 1 ), we have reduced our proof to the following 



Lemma 2.10. Let r = x + w(x,y), where w(x,y) belongs to the ideal 
of A 2 generated by y and w(x,y) ^ K[y], tt the retraction of A 2 onto 
K[r] defined by tt(x) = x + w(x,y), ir(y) = 0, f G K[t] — K. Then tt 
does not preserve the automorphic orbit of f(r). 

Proof. Suppose on the contrary, ix preserves the automorphic orbit of 
f(r). Then for any automorphism a of A 2 , 7ra(f(r)) = (3(f(r)) G K[r] 
for some automorphism (3 of A 2 . Note that 7r/?(/(r)) = 7r 2 /9(/(r)) = 
7ra(/(r)) = (3(f(r)). By Theorem 11.41 7r dog( ^ = it is the retraction of 
A 2 onto the retract K[(3(r)] taking (3{r) to (3{r). By hypothesis, ti is 
also a retraction of A 2 onto the retract K[r] taking r to r. This forces 
that (3{r) = cr + d for some c G K*, d G K. We have concluded that 
for any automorphism a of A 2 , there exists some c G iif*, d G -ft", such 
that 7ra(/(r)) = f(cr + d). 

Now we proceed the proof in two cases. 

1. Noncommutative case: A 2 = K(x,y). 

Denote by C the commutator ideal of K{x, y). 

a) If w(x,y) G C, then take a to be the automorphism of K(x,y) 
defined by a(x) = y + x 2 , a(y) = x. Direct calculation shows that 
7ra(/(r)) = f(r 2 + w(r 2 , r)) = f(r 2 ) ^ f(cr + d), a contradiction. 

b) If w(x,y) <£ C, then w a (x,y) = yv(x,y) for some v(x,y) G 
K[x, y] — {0}. Here w a (x,y) G K[x,y] is the image of w(x,y) un- 
der the abelianization from K(x,y) onto if[a;, y]. Let M be a positive 
integer greater than deg(w(x, y)), it is easy to see that x M — y does not 
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divide v(x,y) in K[x,y]. Let a be the automorphism of K(x,y) de- 
fined by a(x) = x, a(y) = y + x M . Then na(f(r)) = f(r + w(r,r M )) = 
f{r + r M v{r,r M )). As x M — y does not divide v(x,y), v(r,r M ) ^ 0. 
Therefore na(f(r)) = f(r + r M v(r,r )) 7^ /(cr + d), a contradiction. 

2. Polynomial case: A 2 = K[x,y]. 

In this case we write w(x,y) = yq(x,y) where q(x,y) ^ K[y\. Let 
M be a positive integer greater than deg(g(x, y)), it is easy to see that 
x M — y does not divide q(x, y) in K[x, y\. Let a be the automorphism 
of K[x, y] defined by a(x) = x, a(y) = y + x M . Then easy calculation 
shows that 7ra(f(r)) = fir + r M qir,r M )). As x M — y does not divide 
qix,y), qir,r M ) 7^ 0. Therefore 7ra(/(r)) = fir + r M qir,r M )) 7^ /(cr + 
rf). The contradiction completes the proof. □ 
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